Sub-cluster recovery using a partition group index

ABSTRACT

Embodiments disclosed herein provide systems, methods, and computer readable media for sub-cluster recovery in a data storage environment having a plurality of storage nodes. In a particular embodiment, the method provides scanning data items in the plurality of nodes. While scanning, the method further provides indexing the data items into an index of a plurality of partition groups. Each partition group includes data items owned by a particular one of the plurality of storage nodes. The method then provides storing the index.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/077,245 by Shekhar et al., entitled “SUB-CLUSTER RECOVERY USING APARTITION GROUP INDEX,” filed Oct. 22, 2020; which is a continuation ofU.S. patent application Ser. No. 15/443,721 by Shekhar et al., entitled“SUB-CLUSTER RECOVERY USING A PARTITION GROUP INDEX,” filed Feb. 27,2017, now U.S. Pat. No. 10,852,998, issued Dec. 1, 2020; which claimspriority to U.S. Provisional Patent Application No. 62/299,884 byShekhar et al., entitled “SUB-CLUSTER RECOVERY USING A PARTITION GROUPINDEX,” filed Feb. 25, 2016; each of which is assigned to the assigneehereof and hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

Aspects of the disclosure are related to the recovery of data items in adatabase and, in particular, to recovery of data items on a partitiongroup basis.

Overview

Embodiments disclosed herein provide systems, methods, and computerreadable media for sub-cluster recovery in a data storage environmenthaving a plurality of storage nodes. In a particular embodiment, themethod provides scanning data items in the plurality of nodes. Whilescanning, the method further provides indexing the data items into anindex of a plurality of partition groups. Each partition group includesdata items owned by a particular one of the plurality of storage nodes.The method then provides storing the index.

In another embodiment, a system is provided having one or more computerreadable storage media and a processing system operatively coupled withthe one or more computer readable storage media. Program instructionsstored on the one or more computer readable storage media, when read andexecuted by the processing system, direct the processing system to scandata items in the plurality of nodes and, while scanning, index the dataitems into an index of a plurality of partition groups. Each partitiongroup comprises data items owned by a particular one of the plurality ofnodes. The instructions further direct the processing system to storethe index.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the disclosure can be better understood with referenceto the following drawings. While several implementations are describedin connection with these drawings, the disclosure is not limited to theimplementations disclosed herein. On the contrary, the intent is tocover all alternatives, modifications, and equivalents.

FIG. 1 illustrates a computing environment for sub-cluster recoveryusing a partition group index.

FIG. 2 illustrates a method of operating the computing environment toperform sub-cluster recovery using a partition group index.

FIG. 3 illustrates a method of operating the computing environment toperform on-the-fly deduplication.

FIG. 4 illustrates a method of operating the computing environment toperform on-the-fly deduplication.

FIG. 5 illustrates an index system for sub-cluster recovery using apartition group index.

DETAILED DESCRIPTION

In many data storage systems, database storage system in particular,data items are stored across multiple storage nodes. These nodes may beremotely located from one another to provide data redundancy and quickeraccess from different locations. In some storage arrangements, such as aCassandra database, ownership of data items is distributed across someor across all of the storage nodes. As such, data items stored on eachparticular storage node are said to be owned by that storage node.Additionally, some databases feature a concept whereby data ispartitioned into groups that may span physical nodes. For instance, onegroup of data may be stored on four nodes of a physical node cluster.The data in that one group may be considered a single partition groupeven though it is replicated across the four nodes. These partitiongroups are called virtual nodes, or vnodes, in a Cassandra database butthe concept of partition groups may apply to other database types.

When backing up, versioning, or otherwise capturing the state of thestorage nodes in a cluster, storing the data of a particular partitiongroup once allows for deduplication of that data so that the same datais not stored again for each of the storage nodes having a copy of thedata. FIGS. 3 and 4 below describes on-the-fly deduplication during datamovement in a NoSQL data store and the storage of data in partitiongroups, as described in FIGS. 1 and 2 , is one example of how suchdeduplication may be achieved.

FIG. 1 illustrates computing environment 100 for sub-cluster recoveryusing a partition group index. Computing environment 100 includes indexsystem 101, secondary data repository 105, and database nodes 102-104.Index system 101 and database nodes 102-104 communicate overcommunication links 111-113, respectively. Index system 101 andsecondary data repository 105 communicate over communication link 114.In operation, database nodes 102-103 are physical storage systems thatimplement a distributed database that stores data 121-123. In thisexample, data 121 is organized as one partition group 131 in thedatabase, data 122 is another partition group 132, and data 123 isanother partition group 133. Secondary data repository 105 is used byindex system 101 in examples where index system 101 protects data121-123 by backing it up to secondary data repository 105.

FIG. 2 illustrates method 200 of operating the computing environment toperform sub-cluster recovery using a partition group index. Method 200provides index system 101 scanning data items in the data stored ondatabase nodes 102-104 (201). The scanning may occur at any time. Insome cases, the scanning occurs for purposes other than just thosedescribed below for method 200. For example, index system 101 may alsobe configured to backup, version, or otherwise protect the contents ofdatabase nodes 102-104. As part of such a process, data within each ofnodes 102-104 may be scanned in order for data deduplication to beperformed since the same data items may exist across multiple nodes andit is inefficient to store those data items more than once for backuppurposes. Given that scanning is already performed for deduplication,the additional processing resources needed to perform the remaininglimitations of method 200 are relatively negligible.

As alluded to above, while index system 101 is scanning, method 200provides index system 101 indexing the data items into an index ofpartition groups (202). In this example, the three partition groups131-133 are indexed with one of database nodes 102-104 and index system101 assigns as the owner of a particular partition group even thougheach of database nodes 102-104 is an owner of the data items therein.Specifically, partition group 131 includes data 121 and is owned bydatabase node 102, partition group 132 includes data 122 and is owned bydatabase node 103, and partition group 133 includes data 123 and isowned by database node 104. Moreover, the index may further indicatewhich of nodes 102-104 also own the data in each partition group. Whilepartition groups 131-133 are shown as blocks for illustrative andconceptual purposes, it should be understood that the index of the datain partition groups 131-133 can take any form, such as a table, linkedlist, or other type of structure or unstructured data.

After creation of the index, method 200 provides index system 101storing the index (203). The index may be stored to a storage systemwithin index system 101, within secondary data repository 105, or withinsome other storage system. For example, the index may be stored inassociation with a version of database nodes 102-104 that was created atthe same time as the index (i.e. the scanning step was performed to bothcreate the version and the index). The index can then be accessed at alater time in the event that one of database nodes 102-104 needs to berestored or the data in one of partition groups 131-132.

In a particular example of method 200 as applied to a Cassandradatabase, the database maintains sorted string tables (SSTables) to mapthe data stored in database nodes 102-104. Those tables already includeone index of the data in the table. However, the index created above atstep 202 may be an additional index created for each SSTable thatindexes data in the SSTable based on the partition group in which thedata is included. The resource usage cost of creating this additionalindex for each SSTable is negligible because, as noted above, eachSSTable already has data items indexed per row of the table. The indexcreated at step 202 can be done by processing that data items index atstep 201 to scan the data items in the SSTable, which may already beperformed for the purpose of data deduplication or for some otherreason.

Using the partition group index created in step 202, data can berecovered on a partition group basis rather than on a physical databasenode basis. The index is used to identify data for a specific partitiongroup. For instance, instead of using the normal SSTable index toidentify all data on database node 102, the partition group index isused to identify only data 121 that is part of partition group 131. Thepartition group index may further be used to identify data fromadditional partition groups that are also stored on database 102.Regardless, when data of a partition group is to be restored onto adatabase node, the data restoration system (which may be index system101) takes the data items identified from the partition group index forthe partition group and fetches those data items from storage (e.g. froma secondary storage repository). Those data items are then formattedinto a format (e.g. recovery file) that can be understood by thedatabase under which database nodes 102-104 is operating. Many databasesystems support tools that perform such formatting and thereby allowsfor the direct generation of files in a format in which the databasestores. That file can then be used to populate the database node. Forexample, the data items in data 121 may be identified as being inpartition group 131 from the partition group index. Those data items areformatted into a recovery file and the file is used to repopulatedatabase node 102 should database node 102 ever require recovery.

Advantageously, by recovering data on a partition group basis, the datain any particular partition group can be restored to a physical databasenode in one relatively quick file action. Otherwise, the data itemswould have to be repopulated individually onto the database node by thedatabase itself, which could take a large amount of time depending onthe amount of data items due to the need to replicate each record, gothrough different layers, network delay etc. Additionally, therestoration examples above refer to the restoration of data to adatabase node in a production cluster of nodes (e.g. database nodes102-104). However, the partition group index could be used in isolationfrom the production node cluster since, once the recovery file isgenerated that file simply needs to be placed in the desired locationwhether or not the location is in the production cluster. In this way,the production cluster will continue processing requests for other dataitem tables and will not get affected due to recovery of a particulardata item table onto an independent node.

The latter example above allows for efficiency in recovering datacollected from the original cluster to a different sized cluster, as maybe the case when a user, or user group (e.g. business team), wants torecover data from production cluster and want to perform some specificjob on the data. For example, if an analytical team wants to run someexperiment on production data backed up at a particular time. Theembodiments above allow the team to use an experimental cluster, not theproduction cluster, to experiment on that backed up data. Morespecifically, they allow the team to quickly load data into theexperimental cluster, perform the experiment, and then release thecluster, which is advantageous since that cycle will happen many timesfor experimentation. Users can easily create a partition mapping(partition cluster to actual nodes) and, using the partition groupindex, efficiently create the database files. An assumption is that thedatabase will expect partition mapping information from the user, whichdatabases like Cassandra do.

Using computing environment 100 as an example, database nodes 102-104comprise the production cluster of a database consistent with theembodiments above. If a user wanted to experiment on a backup of data121-123 in an experimental cluster, that user may user two databasenodes to instantiate the experimental cluster, as opposed to the threenodes that comprise the production cluster. A first of the two nodes mayinstantiated with data 121 from partition group 131 and data 122 frompartition group 132 while the second of the two nodes includes only data123 from partition group 133. The data for each partition group istherefore only maintained on one of the two nodes in the experimentalcluster, which will only be used for the experiment and will berelatively short lived.

FIG. 3 illustrates method 300 of computing environment 100 forperforming on-the-fly deduplication during movement of NoSQL data. Inparticular, method 300 describes in more generic terms how index system101 can deduplicate data while concurrently indexing the data. Sincededuplication of data conserves storage space by storing a single copyof a duplicated data item, deduplication is a common operation performedwhen storing data, whether that data is being stored in a versioningprocess or otherwise. Performing the deduplication at the same time asthe indexing of method 200 allows index system 101 to conserveprocessing resources by only needing to scan data 121-123 once.

Method 300 provides index system 101 identifying data items from data121-123 during scanning step 201 of method 200 (step 301). The dataitems may be any type of information that is capable of being stored ina file, such as table entries, records, media, and the like, and eachfile may contain any number of data items. The data items may compriseall of the data items stored in data 121-123 or may be only a portion ofthe data items stored in data 121-123. For example, if the data items indata 121-123 are being protected (e.g. backed up), then the data itemsmay comprise only data items that have changed since a previous backup.

Method 300 further provides index system 101 identifying duplicate dataitems from the first data items (step 302). The duplicate data items maybe identified by comparing each of the data items against other ones ofthe data items, by comparing hashes of each of the data items againsthashes of the other ones of the data items, or by some other means ofidentifying duplicate data items.

Once duplicate data items are identified, method 300 provides indexsystem 101 deduplicating and repackaging each of the duplicate dataitems into respective deduplicated data units (step 303). Eachdeduplicated data unit comprises a data form that at least contains botha single instance of the deduplicated data item and informationdescribing the multiple locations (e.g. particular files, positionwithin files, etc.) from which the deduplicated data item originated indata 121-123. The information can then be used should the deduplicateddata item need to be restored, or otherwise, accessed from secondaryrepository in one of its original file locations in files 1-N.

After generating the deduplicated data units, data transfer system 102transfers the deduplicated data units to secondary data repository 105(step 304). Other unique, non-deduplicated data items of the identifieddata items are also transferred to secondary data repository 105. Insome cases, the both the unique data items and the deduplicated dataunits are organized into a file and that file is what is transferred tosecondary data repository 103. Each deduplicated data unit may includeone or more deduplicated data items.

FIG. 4 illustrates operation 400 of computing environment 100 forcreating a deduplicated data unit. In operation 400, 12 data items havebeen extracted from data 121-123 with 10 of those data items beingunique. In one example, the 12 data items are identified during scanningphase of method 200 (i.e., step 201) so that the processing resourcesused during the scanning phase can index data items as well as identifyduplicate data items for the purposes of operation 400. For example, ifdata 121-123 are files in a Cassandra database, then each of data121-123 are parsed to extract the 10 individual items. Each file maycorrespond to and include only 1 data item, although, files in Cassandracan include multiple data items. Thus, it is possible for a single fileto include all the data items in FIG. 4 . Alternatively, if files 1-Nare part of a Mongo database, then the data items within two or morefiles may all be identical at substantially the same time (e.g. even ifat one instant one of the files has more or less data items, the otherfile(s) will eventually catch up). In these cases where files and dataitems therein are identical, the deduplication process need only look atwhether the files themselves are identical to determine that the dataitems therein are also identical.

At step 1, duplicate data items within the 12 extracted data items areidentified. In this example, there are three duplicate instances of dataitem 2. These duplicate instances may be from the same file or may befrom different files. Likewise, the multiple instances of data item 2may be stored across multiple nodes of a data store (e.g., acrossdatabase nodes 102-104). Thus, information regarding duplicate item 2 isexchanged among the data store nodes to determine whether the degree ofduplicates reaches a pre-defined consistency level. That is, if theduplicates do not reach the predefined consistency level, then they arenot deduplicated for the current version. Those duplicates become theleftover for the next version to deduplicate. If more duplicates appearin the next version so that the total number of duplicates reaches theredefined consistency level, the data will deduplicate in the nextversion. In contrast, if the consistency level is met, then theoperation continues as follows. To distribute the work need to determinethe degree of duplicates, data may be partitioned based on keys and eachdata store node may be owners of one or more partitions. Collectingcopies of the same data items (e.g. data item 2) is performed todetermine whether enough copies are present in the database to warrantdeduplication. That is, the resources needed to transfer and store thenumber of copies in secondary data repository 105 are balanced with thetime and resources needed to deduplicate those duplicate data items.

Should the number of duplicate data items 2 be enough to warrantdeduplication in the current version, step 2 repackages the deduplicateddata items into a deduplicated data form. Specifically, found duplicatesare removed and re-organize the remaining unique data items into file402, which includes the remaining unique data items and any informationneeded to restore each copy of item 2. In other examples, the uniquedata items may be organized into more than one file. For a Cassandradatabase, step 2 repackages the remaining unique items (e.g.deduplicated items 1 and 3-10 along with deduplicated item 2) intoSSTables. A Mongo database does not require similar repackaging afterdeduplicating a data item. Once the items have been packaged into file402, file 402 is transferred to and stored in secondary data repository105 at step 3. In some cases, file 402 comprises a data version, such asthe data version of the indexed data discussed above in method 200.

Advantageously, the technology described above allows data 121-123 to bescanned once and produce two outputs that rely on that scanning. Thatis, data 121-123 is scanned to create an index and to deduplicate dataitems therein. The timing of the scanning described herein allows asingle scan to satisfy the needs of both index creation anddeduplication of data items rather than scanning data 121-123 separatelyfor each output.

Referring back to FIG. 1 , index system 101 comprises a computer systemand communication interface. Index system 101 may also include othercomponents such as a router, server, data storage system, and powersupply. Index system 101 may reside in a single device or may bedistributed across multiple devices. Index system 101 could be anapplication server(s), a personal workstation, or some other networkcapable computing system—including combinations thereof. While shownseparately, all or portions of index system 101 could be integrated withthe components of at least one of database nodes 102-104.

Database nodes 102-104 each comprise one or more data storage systemshaving one or more non-transitory storage medium, such as a disk drive,flash drive, magnetic tape, data storage circuitry, or some other memoryapparatus. The data storage systems may also include other componentssuch as processing circuitry, a network communication interface, arouter, server, data storage system, user interface and power supply.The data storage systems may reside in a single device or may bedistributed across multiple devices.

Secondary data repository 105 comprises one or more data storage systemshaving one or more non-transitory storage medium, such as a disk drive,flash drive, magnetic tape, data storage circuitry, or some other memoryapparatus. The data storage systems may also include other componentssuch as processing circuitry, a network communication interface, arouter, server, data storage system, and power supply. The data storagesystems may reside in a single device or may be distributed acrossmultiple devices.

Communication links 111-114 could be internal system busses or usevarious communication protocols, such as Time Division Multiplex (TDM),Internet Protocol (IP), Ethernet, communication signaling, Code DivisionMultiple Access (CDMA), Evolution Data Only (EVDO), WorldwideInteroperability for Microwave Access (WIMAX), Global System for MobileCommunication (GSM), Long Term Evolution (LTE), Wireless Fidelity(WIFI), High Speed Packet Access (HSPA), or some other communicationformat—including combinations thereof. Communication links 111-114 couldbe direct links or may include intermediate networks, systems, ordevices.

FIG. 5 illustrates index system 500. Index system 500 is an example ofindex system 101, although system 101 may use alternativeconfigurations. Index system 500 comprises communication interface 501,user interface 502, and processing system 503. Processing system 503 islinked to communication interface 501 and user interface 502. Processingsystem 503 includes processing circuitry 505 and memory device 506 thatstores operating software 507.

Communication interface 501 comprises components that communicate overcommunication links, such as network cards, ports, RF transceivers,processing circuitry and software, or some other communication devices.Communication interface 501 may be configured to communicate overmetallic, wireless, or optical links. Communication interface 501 may beconfigured to use TDM, IP, Ethernet, optical networking, wirelessprotocols, communication signaling, or some other communicationformat—including combinations thereof.

User interface 502 comprises components that interact with a user. Userinterface 502 may include a keyboard, display screen, mouse, touch pad,or some other user input/output apparatus. User interface 502 may beomitted in some examples.

Processing circuitry 505 comprises microprocessor and other circuitrythat retrieves and executes operating software 507 from memory device506. Memory device 506 comprises a non-transitory storage medium, suchas a disk drive, flash drive, data storage circuitry, or some othermemory apparatus. Operating software 507 comprises computer programs,firmware, or some other form of machine-readable processinginstructions. Operating software 507 includes scan module 508 and indexmodule 509. Operating software 507 may further include an operatingsystem, utilities, drivers, network interfaces, applications, or someother type of software. When executed by circuitry 505, operatingsoftware 507 directs processing system 503 to operate index system 500as described herein.

In particular, scan module 508 directs processing system 503 to scandata items in a plurality of nodes. While scanning, index module 509directs processing system 503 to index the data items into an index of aplurality of partition groups. Each partition group comprises data itemsowned by a particular one of the plurality of storage nodes. Indexmodule 509 further directs processing system 503 to store the index.

The above description and associated figures teach the best mode of theinvention. The following claims specify the scope of the invention. Notethat some aspects of the best mode may not fall within the scope of theinvention as specified by the claims. Those skilled in the art willappreciate that the features described above can be combined in variousways to form multiple variations of the invention. As a result, theinvention is not limited to the specific embodiments described above,but only by the following claims and their equivalents.

What is claimed is:
 1. A method for sub-cluster recovery in a datastorage environment having a plurality of nodes, the method comprising:creating a partition group index that indexes a set of data items into aplurality of partition groups, wherein a partition group of theplurality of partition groups corresponds to a node of a first pluralityof nodes and comprises a subset of data items stored in the node;formatting the set of data items into a recovery file in accordance withthe partition group index; and loading the set of data items included inthe recovery file onto a second plurality of nodes that are configuredto restore the set of data items in accordance with the partition groupindex.
 2. The method of claim 1, further comprising: recovering theplurality of nodes by: identifying the set of data items stored in thenode based on mappings between the plurality of partition groups and thefirst plurality of nodes; and loading the set of data items onto thenode.
 3. The method of claim 1, further comprising: generating per nodedata for one or more nodes of the second plurality of nodes based atleast in part on mappings between the plurality of partition groups andthe first plurality of nodes.
 4. The method of claim 1, wherein thesecond plurality of nodes include fewer nodes than the first pluralityof nodes, the method further comprising: generating a partitiongroup-to-node mapping for the second plurality of nodes from thepartition group index.
 5. The method of claim 1, further comprising:identifying one or more duplicate data items of the set of data items;deduplicating the one or more duplicate data items; repackaging the oneor more duplicate data items into respective deduplicated data units;and storing the respective deduplicated data units to a secondary datarepository.
 6. The method of claim 5, further comprising: determining adegree of duplicates of the set of data items; and comparing the degreeof duplicates to a threshold level of consistency, wherein thededuplication is performed responsive to the degree of duplicates beinggreater than the threshold level of consistency.
 7. The method of claim5, wherein storing the respective deduplicated data units comprises:storing a data version of the set of data items, the method furthercomprising: compiling the respective deduplicated data units into thedata version of the set of data items.
 8. The method of claim 1, whereinthe plurality of partition groups include deduplicated data items. 9.The method of claim 1, wherein the set of data items are stored in a notonly structured query language (NoSQL) data store.
 10. The method ofclaim 1, further comprising: scanning the set of data items stored inthe first plurality of nodes, the first plurality of nodes comprising afirst cluster of nodes; and creating the partition group index whilescanning the set of data items.
 11. The method of claim 1, furthercomprising: storing the partition group index; identifying the set ofdata items included in the plurality of partition groups in accordancewith the partition group index; and instantiating a second cluster ofnodes using the second plurality of nodes based at least in part on thepartition group index.
 12. An apparatus for sub-cluster recovery in adata storage environment having a plurality of nodes, the apparatuscomprising: a processor; memory coupled with the processor; andinstructions stored in the memory and executable by the processor tocause the apparatus to: create a partition group index that indexes aset of data items into a plurality of partition groups, wherein apartition group of the plurality of partition groups corresponds to anode of a first plurality of nodes and comprises a subset of data itemsstored in the node; format the set of data items into a recovery file inaccordance with the partition group index; and load the set of dataitems included in the recovery file onto a second plurality of nodesthat are configured to restore the set of data items in accordance withthe partition group index.
 13. The apparatus of claim 12, wherein theinstructions are further executable by the processor to cause theapparatus to: recover the plurality of nodes by: identifying the set ofdata items stored in the node based on mappings between the plurality ofpartition groups and the first plurality of nodes; and loading the ofdata items onto the node.
 14. The apparatus of claim 12, wherein theinstructions are further executable by the processor to cause theapparatus to: generate per node data for one or more nodes of the secondplurality of nodes based at least in part on mappings between theplurality of partition groups and the first plurality of nodes.
 15. Theapparatus of claim 12, wherein the second plurality of nodes includefewer nodes than the first plurality of nodes, wherein the instructionsare further executable by the processor to cause the apparatus to:generate a partition group-to-node mapping for the second plurality ofnodes from the partition group index.
 16. The apparatus of claim 12,wherein the instructions are further executable by the processor tocause the apparatus to: identify one or more duplicate data items of theset of data items; deduplicate the one or more duplicate data items;repackage the one or more duplicate data items into respectivededuplicated data units; and store the respective deduplicated dataunits to a secondary data repository.
 17. The apparatus of claim 16,wherein the instructions are further executable by the processor tocause the apparatus to: determine a degree of duplicates of the set ofdata items; and compare the degree of duplicates to a threshold level ofconsistency, wherein the deduplication is performed responsive to thedegree of duplicates being greater than the threshold level ofconsistency.
 18. The apparatus of claim 16, wherein the instructions arefurther executable by the processor to cause the apparatus to: store adata version of the set of data items; and compile the respectivededuplicated data units into the data version of the set of data items.19. The apparatus of claim 12, wherein the plurality of partition groupsinclude deduplicated data items.
 20. A non-transitory computer-readablemedium storing code for sub-cluster recovery in a data storageenvironment having a plurality of nodes, the code comprisinginstructions executable by a processor to: create a partition groupindex that indexes a set of data items into a plurality of partitiongroups, wherein a partition group of the plurality of partition groupscorresponds to a node of a first plurality of nodes and comprises asubset of data items stored in the node; format the set of data itemsinto a recovery file in accordance with the partition group index; andload the set of data items included in the recovery file onto a secondplurality of nodes that are configured to restore the set of data itemsin accordance with the partition group index.